{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Contextual RAG**\n",
        "\n",
        "Contextual Retrieval-Augmented Generation (RAG) is an advanced RAG technique that improves response relevance and efficiency by incorporating contextual compression during the retrieval process. Traditional RAG retrieves and sends full documents to the generation model, which may include irrelevant information, leading to higher costs and less accurate responses.\n",
        "\n",
        "In Contextual RAG, the retrieved documents are processed through a Document Compressor before being passed to the language model. This compressor extracts and retains only the most relevant information for the query, or even discards entire irrelevant documents. This approach reduces the noise in the retrieved context, resulting in more precise, concise, and cost-effective responses from the generation model.\n",
        "\n",
        "Reference: [Contextual RAG](https://python.langchain.com/docs/how_to/contextual_compression/)"
      ],
      "metadata": {
        "id": "uSh0jjVbzHCz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Initial Setup**"
      ],
      "metadata": {
        "id": "IDIEkHsh1Eka"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MrZhFJX_yBzX"
      },
      "outputs": [],
      "source": [
        "!pip install --q athina chromadb"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')\n",
        "os.environ['ATHINA_API_KEY'] = userdata.get('ATHINA_API_KEY')"
      ],
      "metadata": {
        "id": "tg6l6UKW06dY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Indexing**"
      ],
      "metadata": {
        "id": "AO7zLKC_1HSm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load embedding model\n",
        "from langchain_openai import OpenAIEmbeddings\n",
        "embeddings = OpenAIEmbeddings()"
      ],
      "metadata": {
        "id": "SS6M8Fvl1HH0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load data\n",
        "from langchain.document_loaders import CSVLoader\n",
        "loader = CSVLoader(\"./context.csv\")\n",
        "documents = loader.load()"
      ],
      "metadata": {
        "id": "iR30q3uZ1YjU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# split documents\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
        "documents = text_splitter.split_documents(documents)"
      ],
      "metadata": {
        "id": "WDEHMq-I1dEE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create vectorstore\n",
        "from langchain.vectorstores import Chroma\n",
        "vectorstore = Chroma.from_documents(documents, embeddings)"
      ],
      "metadata": {
        "id": "cCQ6DKuf1fkf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Retriever**"
      ],
      "metadata": {
        "id": "jC31p6zQ1gRZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create retriever\n",
        "retriever = vectorstore.as_retriever()"
      ],
      "metadata": {
        "id": "fu0TnRi41iUD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Contextual Retriever**"
      ],
      "metadata": {
        "id": "pbNId_f-1u9O"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create llm\n",
        "from langchain_openai import ChatOpenAI\n",
        "llm = ChatOpenAI()"
      ],
      "metadata": {
        "id": "SZfW0rkT2G4x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create compression retriever\n",
        "from langchain.retrievers import ContextualCompressionRetriever\n",
        "from langchain.retrievers.document_compressors import LLMChainExtractor\n",
        "\n",
        "compressor = LLMChainExtractor.from_llm(llm)\n",
        "compression_retriever = ContextualCompressionRetriever(\n",
        "    base_compressor=compressor, base_retriever=retriever\n",
        ")"
      ],
      "metadata": {
        "id": "SXpNy93c3F_Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# checking compressed doc\n",
        "compressed_docs = compression_retriever.invoke(\"what are points on a mortgage\")\n",
        "compressed_docs"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i3r6ULBq1-76",
        "outputId": "c9b64510-3d33-4b19-b3b4-983859a9fefb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. By charging a borrower points, a lender effectively increases the yield on the loan above the amount of the stated interest rate. Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan, thus obtaining a lower monthly payment in exchange for this', metadata={'row': 1, 'source': './context.csv'}),\n",
              " Document(page_content=\"points is the concept of the 'no closing cost loan', in which the consumer accepts a higher interest rate in return for the lender paying the loan's closing costs up front. In some cases a purchaser can negotiate with the seller to get them to pay seller's points which can be used to pay mortgage points.\", metadata={'row': 1, 'source': './context.csv'}),\n",
              " Document(page_content='Points may also be purchased to reduce the monthly payment for the purpose of qualifying for a loan. Loan qualification based on monthly income versus the monthly loan payment may sometimes only be achievable by reducing the monthly payment through the purchasing of points to buy down the interest rate, thereby reducing the monthly loan payment. Discount points may be different from origination fee, mortgage arrangement fee or broker fee. Discount points are always used to buy down the', metadata={'row': 1, 'source': './context.csv'}),\n",
              " Document(page_content='paying points will cost more than just paying the higher interest', metadata={'row': 1, 'source': './context.csv'})]"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **RAG Chain**"
      ],
      "metadata": {
        "id": "znrnrY073o82"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create document chain\n",
        "from langchain.prompts import ChatPromptTemplate\n",
        "from langchain.schema.runnable import RunnablePassthrough\n",
        "from langchain.schema.output_parser import StrOutputParser\n",
        "\n",
        "template = \"\"\"\"\n",
        "You are a helpful assistant that answers questions based on the following context.\n",
        "If you don't find the answer in the context, just say that you don't know.\n",
        "Context: {context}\n",
        "\n",
        "Question: {input}\n",
        "\n",
        "Answer:\n",
        "\n",
        "\"\"\"\n",
        "prompt = ChatPromptTemplate.from_template(template)\n",
        "\n",
        "# Setup RAG pipeline\n",
        "rag_chain = (\n",
        "    {\"context\": compression_retriever,  \"input\": RunnablePassthrough()}\n",
        "    | prompt\n",
        "    | llm\n",
        "    | StrOutputParser()\n",
        ")"
      ],
      "metadata": {
        "id": "DC_S8t0f3pV5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# response\n",
        "response = rag_chain.invoke(\"what are points on a mortgage\")\n",
        "response"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "RnXKAPED3teL",
        "outputId": "478e08df-62ef-4ef1-8edb-2b1c7840cf8d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Points on a mortgage, also known as discount points or mortgage points, are a form of pre-paid interest that borrowers can pay to lenders when arranging a mortgage in the United States. One point equals one percent of the loan amount. By paying points, borrowers can effectively reduce the interest rate on the loan, resulting in a lower monthly payment. Points can also be used to qualify for a loan or to have the lender pay the closing costs upfront. Points are different from origination fees, mortgage arrangement fees, or broker fees. The loan rate is typically reduced by a certain percentage when points are paid.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Preparing Data for Evaluation**"
      ],
      "metadata": {
        "id": "EOz3Wp0Q31ln"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create dataset\n",
        "questions = [\"what are points on a mortgage\"]\n",
        "response = []\n",
        "contexts = []\n",
        "\n",
        "# Inference\n",
        "for query in questions:\n",
        "  response.append(rag_chain.invoke(query))\n",
        "  contexts.append([docs.page_content for docs in compression_retriever.get_relevant_documents(query)])\n",
        "\n",
        "# To dict\n",
        "data = {\n",
        "    \"query\": questions,\n",
        "    \"response\": response,\n",
        "    \"context\": contexts,\n",
        "}"
      ],
      "metadata": {
        "id": "pR8aRn_032H8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create dataset\n",
        "from datasets import Dataset\n",
        "dataset = Dataset.from_dict(data)"
      ],
      "metadata": {
        "id": "NQ_kFqDa4fN2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create dataframe\n",
        "import pandas as pd\n",
        "df = pd.DataFrame(dataset)"
      ],
      "metadata": {
        "id": "YvdWjllI4l5y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "id": "fK4P2QEG4oIu",
        "outputId": "26669cc6-1f72-4aef-d7a9-2a563e4eaa2d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           query  \\\n",
              "0  what are points on a mortgage   \n",
              "\n",
              "                                            response  \\\n",
              "0  Points on a mortgage are a form of pre-paid in...   \n",
              "\n",
              "                                             context  \n",
              "0  [Discount points, also called mortgage points ...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-711f78e3-c458-46cf-913d-78812c7b50f2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>response</th>\n",
              "      <th>context</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>what are points on a mortgage</td>\n",
              "      <td>Points on a mortgage are a form of pre-paid in...</td>\n",
              "      <td>[Discount points, also called mortgage points ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-711f78e3-c458-46cf-913d-78812c7b50f2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-711f78e3-c458-46cf-913d-78812c7b50f2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-711f78e3-c458-46cf-913d-78812c7b50f2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "  <div id=\"id_d946eb4b-ffa2-49e4-9fde-8d26480a2b8e\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_d946eb4b-ffa2-49e4-9fde-8d26480a2b8e button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 1,\n  \"fields\": [\n    {\n      \"column\": \"query\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"what are points on a mortgage\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"response\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Points on a mortgage are a form of pre-paid interest that a borrower can offer to pay a lender in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, a borrower can obtain a lower monthly payment in exchange for this. Additionally, points can also be used to reduce the monthly payment to qualify for a loan. It is important to note that discount points may be different from origination fee, mortgage arrangement fee, or broker fee.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"context\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to dictionary\n",
        "df_dict = df.to_dict(orient='records')\n",
        "\n",
        "# Convert context to list\n",
        "for record in df_dict:\n",
        "    if not isinstance(record.get('context'), list):\n",
        "        if record.get('context') is None:\n",
        "            record['context'] = []\n",
        "        else:\n",
        "            record['context'] = [record['context']]"
      ],
      "metadata": {
        "id": "CjNrTs7l4qg_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Evaluation in Athina AI**\n",
        "\n",
        "We will use **Context Relevancy** eval here. It Measures the relevancy of the retrieved context, calculated based on both the query and contexts. To learn more about this. Please refer to our [documentation](https://docs.athina.ai/api-reference/evals/preset-evals/overview) for further details"
      ],
      "metadata": {
        "id": "JZFlEciE4yFO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# set api keys for Athina evals\n",
        "from athina.keys import AthinaApiKey, OpenAiApiKey\n",
        "OpenAiApiKey.set_key(os.getenv('OPENAI_API_KEY'))\n",
        "AthinaApiKey.set_key(os.getenv('ATHINA_API_KEY'))"
      ],
      "metadata": {
        "id": "vCwCaKjl4ywl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load dataset\n",
        "from athina.loaders import Loader\n",
        "dataset = Loader().load_dict(df_dict)"
      ],
      "metadata": {
        "id": "ivAlmBPc42LB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# evaluate\n",
        "from athina.evals import RagasContextRelevancy\n",
        "RagasContextRelevancy(model=\"gpt-4o\").run_batch(data=dataset).to_df()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 480
        },
        "id": "l3l78eoF43y1",
        "outputId": "11d88637-b234-425a-8f94-279f7bd9c29f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [context_relevancy]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:01<00:00,  1.42s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "You can view your dataset at: https://app.athina.ai/develop/76c73e9b-7e13-4e2e-9cde-37565deefa56\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           query  \\\n",
              "0  what are points on a mortgage   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               context  \\\n",
              "0  [Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan., points is the concept of the 'no closing cost loan', in which the consumer accepts a higher interest rate in return for the lender paying the loan's closing costs up front. In some cases a purchas...   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          response  \\\n",
              "0  Points on a mortgage are a form of pre-paid interest that a borrower can offer to pay a lender in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, a borrower can obtain a lower monthly payment in exchange for this. Additionally, points can also be used to reduce the monthly payment to qualify for a loan. It is important to note that discount points may be different from origination fee, mortgage arrangement fee, or broker fee.   \n",
              "\n",
              "  expected_response             display_name failed  \\\n",
              "0              None  Ragas Context Relevancy   None   \n",
              "\n",
              "                                                                                                                                                                        grade_reason  \\\n",
              "0  This metric is calulated by dividing the number of sentences in context that are relevant for answering the given query by the total number of sentences in the retrieved context   \n",
              "\n",
              "   runtime   model  ragas_context_relevancy  \n",
              "0     1679  gpt-4o                 0.454545  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-32cd7e67-6a5e-4851-b6de-a8c7130fdaf7\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "      <th>expected_response</th>\n",
              "      <th>display_name</th>\n",
              "      <th>failed</th>\n",
              "      <th>grade_reason</th>\n",
              "      <th>runtime</th>\n",
              "      <th>model</th>\n",
              "      <th>ragas_context_relevancy</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>what are points on a mortgage</td>\n",
              "      <td>[Discount points, also called mortgage points or simply points, are a form of pre-paid interest available in the United States when arranging a mortgage. One point equals one percent of the loan amount. Borrowers can offer to pay a lender points as a method to reduce the interest rate on the loan., points is the concept of the 'no closing cost loan', in which the consumer accepts a higher interest rate in return for the lender paying the loan's closing costs up front. In some cases a purchas...</td>\n",
              "      <td>Points on a mortgage are a form of pre-paid interest that a borrower can offer to pay a lender in order to reduce the interest rate on the loan. One point equals one percent of the loan amount. By paying points, a borrower can obtain a lower monthly payment in exchange for this. Additionally, points can also be used to reduce the monthly payment to qualify for a loan. It is important to note that discount points may be different from origination fee, mortgage arrangement fee, or broker fee.</td>\n",
              "      <td>None</td>\n",
              "      <td>Ragas Context Relevancy</td>\n",
              "      <td>None</td>\n",
              "      <td>This metric is calulated by dividing the number of sentences in context that are relevant for answering the given query by the total number of sentences in the retrieved context</td>\n",
              "      <td>1679</td>\n",
              "      <td>gpt-4o</td>\n",
              "      <td>0.454545</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-32cd7e67-6a5e-4851-b6de-a8c7130fdaf7')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-32cd7e67-6a5e-4851-b6de-a8c7130fdaf7 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-32cd7e67-6a5e-4851-b6de-a8c7130fdaf7');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "Out of range float values are not JSON compliant: nan"
            }
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    }
  ]
}